{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Contigency Table for McNemar's Test"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Function to compute a 2x2 contingency table for McNemar's Test"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> `from mlxtend.evaluate import mcnemar_table`    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Overview"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###  Contigency Table for McNemar's Test"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A 2x2 contigency table as being used in a McNemar's Test ([`mlxtend.evaluate.mcnemar`](mcnemar.md)) is a useful aid for comparing two different models. In contrast to a typical confusion matrix, this table compares two models to each other rather than showing the false positives, true positives, false negatives, and true negatives of a single model's predictions:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](./mcnemar_table_files/mcnemar_contingency_table.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For instance, given that 2 models have a accuracy of with a 99.7% and 99.6% a 2x2 contigency table can provide further insights for model selection.\n",
    "\n",
    "\n",
    "![](./mcnemar_table_files/mcnemar_contingency_table_ex1.png)\n",
    "\n",
    "In both subfigure A and B, the predictive accuracies of the two models are as follows:\n",
    "\n",
    "- model 1 accuracy: 9,960 / 10,000 = 99.6%\n",
    "- model 2 accuracy: 9,970 / 10,000 = 99.7%\n",
    "\n",
    "Now, in subfigure A, we can see that model 2 got 11 predictions right that model 1 got wrong. Vice versa, model 2 got 1 prediction right that model 2 got wrong. Thus, based on this 11:1 ratio, we may conclude that model 2 performs substantially better than model 1. However, in subfigure B, the ratio is 25:15, which is less conclusive about which model is the better one to choose."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### References\n",
    "\n",
    "- McNemar, Quinn, 1947. \"[Note on the sampling error of the difference between correlated proportions or percentages](http://link.springer.com/article/10.1007%2FBF02295996)\". Psychometrika. 12 (2): 153–157.\n",
    "- Edwards AL: Note on the “correction for continuity” in testing the significance of the difference between correlated proportions. Psychometrika. 1948, 13 (3): 185-187. 10.1007/BF02289261.\n",
    "- https://en.wikipedia.org/wiki/McNemar%27s_test\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example 2 - 2x2 Contigency Table"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[4, 1],\n",
       "       [2, 3]])"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import numpy as np\n",
    "from mlxtend.evaluate import mcnemar_table\n",
    "\n",
    "y_true = np.array([0, 0, 0, 0, 0, 1, 1, 1, 1, 1])\n",
    "\n",
    "y_mod1 = np.array([0, 1, 0, 0, 0, 1, 1, 0, 0, 0])\n",
    "y_mod2 = np.array([0, 0, 1, 1, 0, 1, 1, 0, 0, 0])\n",
    "\n",
    "tb = mcnemar_table(y_target=y_true, \n",
    "                   y_model1=y_mod1, \n",
    "                   y_model2=y_mod2)\n",
    "\n",
    "tb"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To visualize (and better interpret) the contigency table via matplotlib, we can use the `checkerboard_plot` function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXsAAAFKCAYAAAD8GP8DAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAAPYQAAD2EBqD+naQAAFo5JREFUeJzt3HuwnHWd5/H355gbjGC4SAyXKBQQwyyMJgvCilDDpbQG\nMw5VFmwpDoIuAzKgK2UBI1JDMWwNXoryEm6TRYYJMlpicUt2UCA6hQsTDSYgxEAQAiQjEBJCIIQA\n57d/PE+095iEE9KHPie/96uqq7r7efrp73Oafp+nnz4hpRQkSdu2vl4PIEkaesZekipg7CWpAsZe\nkipg7CWpAsZekipg7CWpAsZekipg7CWpAsZekipg7CWpAsZekipg7CWpAsZekipg7CWpAsZekipg\n7CWpAsZekipg7CWpAsZekipg7CWpAsZekipg7CWpAsZekipg7CWpAsZekipg7CWpAsZekipg7CWp\nAsZekipg7CWpAsZekipg7DUsJLksyTWDXPfkJL8a6pnUXcPtNU4yJ8npg1z3yCSrhnKeoWbsNVKV\njd2ZZEySq5P8NsnqJA8lOeWtHk5dMaSvcSnlL0opV27tPO1Mw/4AZFSvB5C6bBSwHDiqlPJ4kg8A\n/yfJk6WUO7r5RElGl1Je7eY2NShb9RonCZBSSn8XZwqb+WUwHHhkr0FJ8liS85LMS/JiktlJdkoy\nI8mqJIuTHNqx/tvbo6/lSZYluSLJ9h3Lj0hyf5IXkvwQ2GHA8+2T5JYkz7TP/eXBzFlKWVtK+ftS\nyuPt7f8A5gKHb2K/TkhyT8ftG5Ms77j99STfbK9/N8nMJN9P8jxwWnv/OUmWJFnRnhrYe8DP7UtJ\n7mn3dW6SPTqWH9AuW53kziSXJpk7mH3ttm31NR6wb/cALwJT2tfi7I51Pp7kkXZfr05ya5ILB2zn\nM0meSPJskkvb+94HXAEcmGRNu797DmZf3lKlFC9e3vACPAbcB+xO86Z9AFgMfIzmqObvgYUd618D\n3AGMB3ameTNe1S4bD6wCPktzwPFRYB1wTbt8u/b5zgLeBuwJ3A+c0i4/GbhvkHOPA54Ejt/E8t2A\nV4A/aW//DngUmNzevg/4y/b6d2lCcUzHtv+63f4BwBjg68Cvgb6On9sCYFK7fHbHfo4ClgAXtNcP\nAZ4F7vI17t5r3LFvi4B9230Z3c57drt8f2AtcGw772eB9cCF7fIjgdfa13cM8N72v4UjtnTenr2H\nez2Al5Fxad8s/6Pj9qXAzztuT2nfDKPaN9M64L92LD8MWNte/xTw6wHbn9MRgo8D8wcs/yxwR3t9\nS0Iwa8PjNrPOA8BHgPcB/wF8Ezgd2Kl9w7+jXe+7wI8GPPbHwJc6bo8BVgOHbuLn9okNwQQ+BKyk\n/cXQ3vcdehv7bfU1fgw4a8B9nbG/ALhlwPJfbyT2Ywe89v9zS+ft1cVz9toST3dcX7uR2wG2pznS\nGg0s7Vj+W2Bskl2BiQOW0d4e215/D81H4pXt7bSXJ7Zk2CRXAPsBx7zBqj8FjqI5qr8LuAc4CXiG\nJsyrO9YdOMOewOMbbpRS1rengTo/xnf+nF7iD6czJgL/Wf7/c8dP0HxK6JVt9TWG5uh/U3bfyPKB\ns7xQSnml43bnaznsec5eQ+FZmiPi93TctzfwSillBc2Xa+8e8JhJHdefBH5ZStm5vexUShlfSjlo\nsAMkuRw4GDi2lLLmDVafC/x5e7kL+BnNUffR7bJOA7/Ue4qO/Uwyho2HY2OWA+9K0vk+nLSplYeZ\nkfYawx+/dp2WA3ttZt6t2fawYOzVTQEozefaG4BL2i/4dgEuAa5r15sN7NF+2fW2JMfRHFlvcBsw\nIckZScYm6Uuyf5IjBzVEMoPmlMKxpZQXBvGQnwF/1j7m7vZI/imaUy53vcFjZwF/m2RKkrHAP7SP\n/cUgnvde4Hng/CSjkhwMnDCIx/XSSH2N38gPgGOSHNPOeyrNJ4bBehqYmGRcF2YZEsZegzWYPyvr\nXOfzNKc3HqI5J/4wcA5AKWUVzZd+X6D5Eu9UmmjSLn+J5mP50e02VgDXAxPeaIAkk4AzgMnA0o6/\njrh8k0OX8hzwIPBgKeXl9u47ab5E/PfN7nAp1wHfponXcuBAYHrHqZlN/txKKa/R/Bym05y7/0fg\nX2i+MO6FbfY13sS+/f6+UsrDNOfdr2xn+QDNL/rNvRad27yL5pf3siQrh+Nf46T9ckHSMJDkSpr3\n5d/0epbaJfkNcFEp5YZez9INHtlLPZTk8CR7pnE0zamjH/R6rhol+Wj7bwfGJjkHeBfwb72eq1v8\naxypt/YB/pXm79KfAs4tpdzZ25Gq9WHgn2m6uJjmdNyI/v/hdPI0jiRVwNM4klQBYy9JFTD2klQB\nYy9JFTD2klQBYy9JFTD2klQBYy9JFTD2klQBYy9JFTD2klQBYy9JFTD2klQBYy9JFTD2klQBYy9J\nFTD2klQBYy9JFTD2klQBYy9JFRjV6wHUG0kmAbv2eg511Rhgfa+HUNesKKU80a2NpZTSrW1phEgy\nqa+vb3F/f/+4Xs8iaZPWAZO7FXyP7Ou0a39//7hZs2YxZcqUXs+iLpgzZw5f+cpXej2Gumsczadv\nY6+tM2XKFKZOndrrMdQFixYt6vUIGub8glaSKmDsJakCxl6SKmDsJakCxl6SKmDsJakCxl6SKmDs\nJakCxl6SKmDsJakCxl6SKmDsJakCxl6SKmDsJakCxl6SKmDsJakCxl6SKmDsJakCxl6SKmDsJakC\nxl6SKmDsJakCxl6SKmDsJakCxl6SKmDsJakCxl6SKmDsJakCxl6SKmDsJakCxl6SKmDsJakCxl6S\nKmDsJakCxl6SKmDsJakCxl6SKmDsJakCxl6SKmDsJakCxl6SKmDsJakCxl6SKmDsJakCxl6SKmDs\nJakCxl6SKmDsJakCxl6SKjBiY5/ksiTXDHLdk5P8aqhnkqThasTG/k0om1qQ5Mwkv0iyLsmP3sqh\nNLTWrVvHvvvuy84779zrUbQVPve5zzFv3jxefvllbrzxxl6PMyLVFPvNWQZcDFw9lE+SZPRQbl9/\n7MILL2Tvvffu9RjaSsuWLePiiy/m6quH9C26TRuy2Cd5LMl5SeYleTHJ7CQ7JZmRZFWSxUkO7Vj/\n7UmuTrI8ybIkVyTZvmP5EUnuT/JCkh8COwx4vn2S3JLkmfa5vzzYWUspN5VSbgGeG8R+nZDkno7b\nNyZZ3nH760m+2V7/bpKZSb6f5HngtPb+c5IsSbIiyZwke3c8/rEkX0pyT7uvc5Ps0bH8gHbZ6iR3\nJrk0ydzB7mtN5s+fz+233865557b61G0lW6++WZuvfVWnnvuDd+i2oShPrI/AfgrYCIwCbgX+DGw\nM3ADcFXHut8C9gEOAA4E3gtcBpBkPHBzu8544FrgpA0PTLIdcCfwk/a5PgScmOSUIdinnwJTk/xJ\ne/uDwMtJJre3j2pn2eC/A/9UShkP/O8kfw18AfhLYHfgIeDWJJ2vxSeBE4FdgbU0nzpIMgq4BZgN\n7AKcD5zKZk5R1er111/ntNNO4/LLL2f0aD9QSUMd+ytKKctLKWuAOcCKUsrNpZQCfB/40ySjkgT4\nBHBeKeX5UspK4O+AT7XbmQ4sK6XMLKX0l1JuA+7qeJ7jgJWllG+XUl4vpTxF84vhk93eoVLKM8DD\nwIeSvA9YCtwG/HmSnYD/Avys4yE/LqXc0T52Hc0vqW+VUh4qpaxv93Mv4JCOx8wopTzRLr8emNbe\nfxjNL8r/VUp5rZQyj+bnqAG+9rWvMW3aND74wQ/2ehRpWBg1xNt/uuP62o3cDrA9MA4YTRPODX4L\njE2yK83Reucy2ttj2+vvAQ5MsrK9nfbyxNbvwkb9lOYI/nc0v3TuoYn4M8DCUsrqjnUHzrAn8PiG\nG6WU9e1poD071un8Ob3EH05ZTQT+s5TSP2D7B7zZHdkWPfroo1x55ZUsWLAAgObYQqrbUMd+sJ4F\n1tNE+9n2vr2BV0opK9oYvnvAYybxhyg+CfyylPLf3oJZAebSnEL5Hc0niHk0p6SebZd16h9w+yma\n/QQgyRia0zlPDuJ5lwPvStLXEfxJWzr8tu7uu+/mmWeeYf/996eUwquvvsqaNWvYbbfdmD17Ngcf\nfHCvR5Tecr3+a5wAtKd1bgAuab/E3QW4BLiuXW82sEeSzyR5W5LjaI6sN7gNmJDkjCRjk/Ql2T/J\nkYMaotnmhk8Xfe02Nnei92fAn9GcVrm7PZJ/iuZU1F2beRzALOBvk0xJMhb4h/axvxjEqPcCzwPn\nt6e/Dqb5XkQdTjzxRJYsWcKCBQtYuHAhM2fOZMcdd2ThwoW8//3v7/V4ehP6+voYO3Yso0ePpq+v\njzFjxjBq1HA5Vh0ZhjL2g/ns3LnO52lObzwEPEBzXvwcgFLKKuBjNF9srqL5UnLW7zdSykvAMcDR\n7TZW0JzrnjDIWS+gOa10Ps33A2uB2zc5dCnPAQ8CD5ZSXm7vvhPYDvj3zT1RKeU64Ns0v6CW03wZ\nPb3jSH2TP7dSyms0P4fpwErgH4F/AV7Z/O7VZdy4cey+++6/v7zzne8kCRMnTjQQI9QFF1zA2rVr\nOf/885k+fTpr167l9ts3+RbVRsTzmSNbkitpXse/2YLHTAXmz58/n6lTpw7dcHrLXH/99Zx00klv\nvKJGmmmllPu6saFen8bRFkpyeJI90zia5tTRD3o9l6Thzc+0I88+wL/S/HuDp4BzSyl3bv4hkmpn\n7EeY9pz/dW+4oiR18DSOJFXA2EtSBYy9JFXA2EtSBYy9JFXA2EtSBYy9JFXA2EtSBYy9JFXA2EtS\nBYy9JFXA2EtSBYy9JFXA2EtSBYy9JFXA2EtSBYy9JFXA2EtSBYy9JFXA2EtSBYy9JFXA2EtSBYy9\nJFXA2EtSBYy9JFXA2EtSBYy9JFXA2EtSBYy9JFXA2EtSBYy9JFXA2EtSBYy9JFXA2EtSBYy9JFXA\n2EtSBYy9JFXA2EtSBYy9JFXA2EtSBYy9JFXA2EtSBYy9JFXA2EtSBYy9JFXA2EtSBYy9JFXA2EtS\nBYy9JFVgVK8HUO/MmTOHRYsW9XoMdcHPf/7zXo+gYS6llF7PoLdYkkOBe3o9h7qrr6+P/v7+Xo+h\nLmlfz8NKKfd2Y3se2ddpfa8HUPf19/cza9YspkyZ0utRtJUWLVrESSedBF18rxp7aRsyZcoUpk6d\n2usxNAz5Ba0kVcDYS1IFjL0kVcDYS1IFjL0kVcDYS1IFjL0kVcDYS1IFjL0kVcDYS1IFjL0kVcDY\nS1IFjL0kVcDYS1IFjL0kVcDYS1IFjL0kVcDYS1IFjL0kVcDYS1IFjL0kVcDYS1IFjL0kVcDYS1IF\njL0kVcDYS1IFjL0kVcDYS1IFjL0kVcDYS1IFjL0kVcDYS1IFjL0kVcDYS1IFjL0kVcDYS1IFjL0k\nVcDYS1IFjL0kVcDYS1IFjL0kVcDYS1IFjL0kVcDYS1IFjL0kVcDYS1IFjL0kVcDYS1IFjL0kVWCb\njn2Sy5JcM8h1T07yqyGeZ06S0we57pFJVg3lPJLqsU3H/k0om1qQ5Mwkv0iyLsmP3tTGS/mLUsqV\nXZpnyH85jVSjR4/mqquu4tFHH+X555/nwQcf5NOf/nSvx9JWOvvss5k0aRLveMc72GuvvfjiF7/I\na6+91uuxRgxjP3jLgIuBq7f0gWl0+2cdNvPLoGajRo1i+fLlHHXUUYwfP55TTjmFb3zjGxx99NG9\nHk1b4cwzz2Tx4sWsXr2ahQsXsmDBAr761a/2eqwRo6exT/JYkvOSzEvyYpLZSXZKMiPJqiSLkxza\nsf7bk1ydZHmSZUmuSLJ9x/Ijktyf5IUkPwR2GPB8+yS5Jckz7XN/ebCzllJuKqXcAjy3hft2D/Ai\nMCXJ3CRnd6zz8SSPtPt6dZJbk1w4YDufSfJEkmeTXNre9z7gCuDAJGva/d1zsPuyrXv55Ze56KKL\nWLp0KQDz5s1j7ty5HH744T2eTFtj8uTJbLfddgC8/vrr9PX18cgjj/R4qpFjOBzZnwD8FTARmATc\nC/wY2Bm4AbiqY91vAfsABwAHAu8FLgNIMh64uV1nPHAtcNKGBybZDrgT+En7XB8CTkxyypDtGZwM\nfAp4O/Bw54Ik+wPXAZ8DdgHmAR8e8PgdgSnAvu28ZyY5opSyADgdeKCUskMpZcdSylNDuB8j2tix\nYznkkENYuHBhr0fRVrr00kvZYYcdmDBhAvfffz9nnXVWr0caMYZD7K8opSwvpawB5gArSik3l1IK\n8H3gT5OMShLgE8B5pZTnSykrgb+jiSnAdGBZKWVmKaW/lHIbcFfH8xwHrCylfLuU8nobx28BnxzC\nfbu8lLKkNF4dsOwE4I5Syk/aeWcy4BdC68ullPWllN8A/xeYNoTzbpNmzpzJ4sWLuemmm3o9irbS\nueeey5o1a1i0aBGnn346EyZM6PVII8ZwiP3THdfXbuR2gO2BdwKjgaUdy38LjE2yK83ReucyBtx+\nD81pj5XtZRXwdWC3buzEJjy5mWW7b2T5EwNuv1BKeaXj9ksMODWlzbv88svZb7/9OP7443s9irpo\n8uTJHHTQQX7xvgWGQ+wH61lgPU20N9gbeKWUsgJYDrx7wGMmdVx/EvhlKWXn9rJTKWV8KeWgIZy5\nfzPLlgN7Dbhv0sZWfBPbFjBjxgwOPvhgjj32WF588cVej6MuW79+PUuWLOn1GCPGSIh9ANrTOjcA\nl7Rf4u4CXEJz3htgNrBH+4Xm25IcBxzVsZ3bgAlJzkgyNklfkv2THDmoIZptjqP5dNHXbmP0VuzX\nD4BjkhzTbvtUYL8tePzTwMR2Jg3wne98h8MOO4xjjz2WNWvW9HocbaWXXnqJa6+9ltWrVwPwwAMP\ncMkll/CRj3ykx5ONHL2O/WD+dLBznc8DjwMPAQ/QnOM+B6CUsgr4GPAFYBVwKjDr9xsp5SXgGODo\ndhsrgOuBwZ70u4DmtNL5NN8PrAVuH+Tcf3RfKeVhmi9wr2xn+QDNdwyvbORxG9vmXTRfZi9rT0v5\n1zitvfbaizPOOIPJkyezdOlSXnjhBVavXs2MGTN6PZrepCR873vfY99992XHHXfk+OOPZ/r06Vx2\n2WW9Hm3ESHPArOEgyW+Ai0opNwzx80wF5g/lc6g35s+fz9SpU3s9hrbSfffdx7Rp0wCmlVLu68Y2\ne31kX7UkH23/7cDYJOcA7wL+rddzSdr2jOr1AJX7MPDPNK/DYmB6ezpKkrrK2PdQKeUswH8VImnI\neRpHkipg7CWpAsZekipg7CWpAsZekipg7CWpAsZekipg7CWpAsZekipg7CWpAsZekipg7CWpAsZe\nkipg7CWpAsZekipg7CWpAsZekipg7CWpAsZekipg7CWpAsZekipg7CWpAsZekipg7CWpAsZekipg\n7CWpAsZekipg7CWpAsZekipg7CWpAsZekipg7CWpAsZekipg7CWpAsZekipg7CWpAsZekipg7CWp\nAsZekipg7CWpAsZekipg7CWpAsZekipg7CWpAsZekipg7CWpAsZekipg7CWpAsZekiowqtcDSOqe\nRYsW9XoEdcFQvI4ppXR9oxrekkwCFgPjej2Luqevr4/+/v5ej6Eu6evrW9ff3z+5lPJEN7Zn7CvV\nBn/XXs+hrhoDrO/1EOqaFd0KPRh7SaqCX9BKUgWMvSRVwNhLUgWMvSRVwNhLUgWMvSRVwNhLUgWM\nvSRVwNhLUgWMvSRVwNhLUgWMvSRVwNhLUgWMvSRVwNhLUgWMvSRVwNhLUgWMvSRVwNhLUgWMvSRV\nwNhLUgWMvSRVwNhLUgWMvSRVwNhLUgWMvSRVwNhLUgX+H4siYm2nMCUGAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x105f785c0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from mlxtend.plotting import checkerboard_plot\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "brd = checkerboard_plot(tb,\n",
    "                        figsize=(3, 3),\n",
    "                        fmt='%d',\n",
    "                        col_labels=['model 2 wrong', 'model 2 right'],\n",
    "                        row_labels=['model 1 wrong', 'model 1 right'])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## API"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "## mcnemar_table\n",
      "\n",
      "*mcnemar_table(y_target, y_model1, y_model2)*\n",
      "\n",
      "Compute a 2x2 contigency table for McNemar's test.\n",
      "\n",
      "**Parameters**\n",
      "\n",
      "- `y_target` : array-like, shape=[n_samples]\n",
      "\n",
      "    True class labels as 1D NumPy array.\n",
      "\n",
      "- `y_model1` : array-like, shape=[n_samples]\n",
      "\n",
      "    Predicted class labels from model as 1D NumPy array.\n",
      "\n",
      "- `y_model2` : array-like, shape=[n_samples]\n",
      "\n",
      "    Predicted class labels from model 2 as 1D NumPy array.\n",
      "\n",
      "**Returns**\n",
      "\n",
      "- `tb` : array-like, shape=[2, 2]\n",
      "\n",
      "    2x2 contingency table with the following contents:\n",
      "    a: tb[0, 0]: # of samples that both models predicted correctly\n",
      "    b: tb[0, 1]: # of samples that model 1 got right and model 2 got wrong\n",
      "    c: tb[1, 0]: # of samples that model 2 got right and model 1 got wrong\n",
      "    d: tb[1, 1]: # of samples that both models predicted incorrectly\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "with open('../../api_modules/mlxtend.evaluate/mcnemar_table.md', 'r') as f:\n",
    "    s = f.read() \n",
    "print(s)"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}
